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Abstract Based on Pontryagin Maximum Principle (PMP), this paper established 
a generalized PMP aiming at non-feedback control system with stochastic initial 
conditions. We proved the conclusion and show its coming back to PMP when the 
randomness collapses. Through the generalized PMP, a general method is intro¬ 
duced to solve expectation maximum problem of these systems and thereafter an 
example showed its feasibility. 
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1 Introduction 


Optimal control is a crucial component of modern control theory. Generally, once 
given initial conditions, cost function and admissible control set, we can use either vari¬ 
ational method or maximum principle [I] to find the optimal trajectory of certain control 
system without stochasticity. The optimal trajectory refers to the one that minimizes the 
cost function. To solve a given optimal control problem, one of the most used methods is 
to transform the system through Pontryagin Maximum Principle (PMP) into Hamilton 
equation|2j. This method leads to predictable results in most cases with no random¬ 
ness. However, stochasticity is not always avoidable in practice. This paper is focused on 
the control systems with stochastic initial conditions. Since a single preinstalled optimal 
control cannot deal with all existing cases, a new method is needed to substitute PMP. 

We consider a generalization of PMP, which replacing the original optimal conditions 
with the expectation of cost function. In detail, we need a new conclusion named PMP* 
satisfying two requirements at least: 

1) Given configuration space M and initial condition space if the stochasticity satisfies 
certain condition, there exists a stochastic Lipschitzian curve A* 6 T*M according to 
PMP*. Besides, the curve satisfies the maximum condition analogously. 

2) When condition space becomes a set of a single point, X* converges to X t given by 
PMP. That is, PMP* returns to PMP. 

When the generalized PMP is attained, generalized Hamilton equation can be con¬ 
structed, and therefore optimal stochastic curve can be found. This paper will later give 
an example to show the feasibility of this method and if it comes back to PMP. 

Acknowledgement: I would like to thank Mr. Zhangju Liu for his useful advice and 
great encouragement. 


2 Generalization of PMP from geometric viewpoint 

We now consider following optimal control problem from geometric viewpoint: M is 
a configuration space, U C R" is admissible control set. q 6 M is the state of system, 
and u(t) EU is a preinstalled control. 

Q = /«(?),?£ M,ueU, (2.1) 

with initial conditions 

<7(0) = <7o, (2.2) 

Given cost function : M x U — Y R, the problem is to find a control u = u that minimizes 

J= [ <p(q(t),u(t))dt. 

Jo 

We define Hamiltonian as h u ( A) = (A,/ U (A)) and thereupon have the following geo¬ 
metric statement of PMP: 
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Theorem 2.1 (PMP). Given an admissible control u = u{t), if (2.1), (2.2) have a solu¬ 
tion q(t ) = g fi ( t )(f), if 


q(ti) G dA qo (ti), 

here A qo (ti) indicates the attainable set from point q 0 in t\ time. 

Then there exists a Lipschitzian curve X t G TMM, t G [0, ti] in the cotangent bundle, 
such that 


^ o, 

(2.3) 

Xt hu( t )(X t ), 

(2.4) 

hz(t)(X t ) = m&x{h u n){X t ), 

u£U 

(2.5) 


for almost all t G [0, , ti] 

The proof can be found in [3]. 

Now we define right side of (2.2) as a random variable in probability space (12, F, P), 
rather than a fixed value in R. The probability density is hereinafter denoted as f(x). 

The problem is to find the optimal control u minimizing cost expectation J = 
Ej* 1 ip(q{t),u(t))dt 

Then we have the following conclusion 

Theorem 2.2. [PMP*]Given an admissible control u = u{t), if (2.1), (2.2) with a stochas¬ 
tic q 0 have a solution q(t ) = g«(p(t), if 

q(G) G dAf 0 (t i), 

here A qo (ti) indicates the expected attainable set from point q 0 in t\ time. That is, 

A x( t ) = {[ a x f(x)dx\a x G A x (t)} (2.6) 

J M 

Meantime, u is injection: Vw, q^ ^ q^\ i.e.q^ ^ q ^ 

Then there exists a stochastic Lipschitzian curve \* t G TM M x (12,F, P),t G [0,£i], 
such that 


K # 0, (2.7) 

K = hu(t){ K), (2-8) 

Ehu(t)(K) = max Eh u ^ (A*), (2.9) 


for almost all t G [0,fi] 

Proof: To prove the conclusion, a vector field depending on two parameters needs to 
introduced first: 


9t,u P t *(Ai(t) fu(r))M ^ [0,fl],W ^ U 


According to [1], 


q u {ti) = qi ° exp f g T , u{r)dx 
Jo 


Thus we have the following lemmas: 


3 


Lemma 2.3. Let V £ [0,£i] be the set of Lebesgue points of the control «(•). If 


T qi M = cone{g T)U {qi)\r £ P,u £ P} 


Then 

qi e intA qo (ti) = A go (£i)\M go (£i) 

The proof is in [1] 

Lemma 2.4. Let T £ [0,ti] be the set of Lebesgue points of the control u(-). If 
T qi M = cone{g TjU (qi)\T £ T, u £ P},Vgi G Oj = {gi|g 0 £ fi} 


Then 

Eq i e intAgih) = Agft^XdAgfa) 

Proof: For any qf° £ Oi, Let O g 9o C intA qo {tf) be its neighbourhood. Since intA qo is 
open for any q 0 , the neighbourhoods are able to be attained. Consider the following set: 

Og(t) = { [ a q J(x)dx\a qo £ O qo (t), q 0 £ 12} 

J M 

By Lemma [2.31 we have 

Eqi = f q\f{x)dx £ Og(t) C Ag{t x ) 

J M 

Since Og(t) is open, there must be Eq\ £ intAgiftf). ■ 

Now return to the proof of theorem 12.21 The terminal point of stochastic curve A* 
satisfies Ag(t) = {intMd x f( x )dx\a x £ A x (t)}. By Lemma 12.41 if this condition holds, 
Vgi £ f2i,a.e., origin point o qi £ T qi M belongs to dcone{g T ^ u (qi)\r £ T,u £ U}. Notice 
that cone{g TtU (qi)\T £ T, u £ U} is an open set. thus, 

3a;£t;x(o,f,p),a;^o, 

satisfies: 

E(\* tl , gt,u(qi)) <0,V££ [0, £i], n £ Ua.e. 

that is 
further, 

E(Pl'‘X h ,Uqi)) < /„(?,)> 

Then flow P/ 1 defines stochastic curve Ajf in T*^ x (Q,F, P): 

\;&Pt"\- h eT? m x(n,F,p) 

Injection condition ensures that the curve is well-defined. In terms of this covector 
curve, the inequation above reads: 

E{K,f u (qi))<E{K,h(qi)) 

which exactly equals to Hamilton maximum conditions: 

EK(t){K) = maxP{/i u( i)(A*) 
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Besides, since the curve is fixed once the terminal point is fixed, the following equation 
holds: 

/ t\ rt i ^ 

fu{x)dx)* = X tl O (exp J hu( x )dx ) 

i.e. 

'dqii 

that is, 

V<ZiA* = /^ (t) (A t *), 

Thus, the existence of extremal stochastic curve is proved. ■ 


3 Application of generalized PMP in optimal control 
problems with stochastic initial conditions 


3.1 Statement of problem 

Before applying PMP* into practice, we shall first make clear which problems are 
adaptive to it and which are not. 

PMP is used to solve optimal control problems which have strict limitation for termi¬ 
nal point. However these limitations cannot be satisfied in stochastic control systems by 
a single preinstalled control. If vague limitation is required for terminal point, a penalty 
function I(x),x G M can be introduced to measure if a point is close enough to our 
expectation. 

Thereout, we can define new cost function J(u) = E J* 1 ip(q(t),u(t))dt + EI(q(ti)). 
Let I(x) be smooth, define 

<p(q(t),u(t)) = (p(q(t),u(t)) + (DI(q(t),u(t))) ■ f u (q(t)), 


thus we get 


J = E 


<p(q(t),u(t))dt. 


(3.1) 


^0 

The problem has been transformed into a optimal control problem with stochastic 
initial conditions and no limitation to terminal conditions. The section is focused on 
problems of this kind. 

Consider the following optimal control problem: 


and the cost function is defined as 


We extend this control system as follows: 

q = f A>( m ) 


q = f u {q),Q eM,ueU 

(3.2) 

q(0) = q 0 E (ft, F, P ) 

(3.3) 

ti fixed 

(3.4) 

s 

rii 

= E <p(q(t),u(t))dt , 

Jo 

(3.5) 


5 



and extend the corresponding vector field: 


<p{q, u ) 

fu(q) 


fu(q) = ( 

Thus we get a new control system: 


q = fu(q), qeM,ueU 

(3.6) 

m =*=(“) 

(3.7) 

tifixed 

(3.8) 


where go is a (n+l)-dimension random vector. 

Remark 3.1. Notice that if u is optimal control, then the following condition holds: 

Ehi( t )( A*) = maxE{h u ^(\* t ) 

uEU 

Then the terminal point satisfies: 

q(ti) e dAf o ( tl ), 

Therefore, PMP* can be applied to find the extremal curve. 


3.2 Introduction of extremal parameter 

When use PMP* to solve optimal control problems, we need transform it into Hamil¬ 
ton equations through the Hamiltonian. However, directly using the previous Hamiltonian 
does not extinguish maximum from minimum of cost function. To make up the defect, 
we introduce new parameters v and a new control w. Define y = J qo {u ), and consider the 
following system: 


V = p(q,u) + w (3.9) 

q = fu(q) (3.10) 

Then the extremal stochastic curve in origin system corresponding to the control 
w(t) =0. As a result, it comes to the boundary of attainable set at t\. Apply PMP to it. 
Define new Hamiltonian as follows: 


h( w , u )(iy, A*) = (A*, f u ) + u(<p + w) 
The corresponding Hamilton system is 

dv dEh 

= 0 


dt dy 

dEy p , 

A* = hu(t){\) 


The first equation stands for v = constant. 


(3.11) 


(3.12) 
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In terms of this system, Hamilton Maximum condition is: 

E((Xt,fu(t)) + u<p(q(t),u(t))) = max E((X*,f u{t) ) + v<p(q(t),u(t))) + uw 

u€:U,w>[j 

Since the maximum of original system is attained, there must be v < 0, thus we can set 
v = 0 in the right hand of maximum condition: 

E((X* t , kit)) + v<p(q(t),u(t))) = max E((X* t , f u (t)) + v<p(g(t),u(t))) 

u£U,w> 0 

So we prove the following conclusion: 

Theorem 3.2. If u = u(t) ts the optimal control for problem (3.9)-(3.11), that is, u(t) 
minimizes J (u). Define generalized Hamiltonian family: h”( A) = (A*, fu(t))+v<p(q(t), u(t)), 
X t € TfM x (Q,F,P),t G [0,ti],q G M, u G JJ, u G Rthen there exists a stochastic Lips- 
chitzian curve X* G T~^M x (fl,F,P),t G [0, £i], and a number v G R such that 


(x;,u)^o, (3.13) 

K = hir{t)(K), ( 3 - 14 ) 

E K(t)(X* t ) = max^ w (A*), (3.15) 

// < 0, (3.16) 


for almost all t G [0, tf 

Remark 3.3. since pair (A*,z/) can 6e multiplied by any positive number, only abnormal 
case v = 0 and normal case v — —l need to be considered. Besides, when solving maximum 
problems, u < 0 becomes v > 0, that is, analogously abnormal cases v = 0 and normal 
case v = 1. 


3.3 Example 

To demonstrate this method more specifically, we consider its application to classic 
Cheapest Stop Problem. 

The original problem can be described as follows: A train moves on the railway. We 
start braking the train at certain initial location and speed. The goal is to stop it with 
minimum expenditure of energy, which is assumed proportional to the integral of squared 
acceleration. 

Its mathematic statements is 

x — u, 
x(0) = Xq 
x(0) = v 0 

ti fixed, x(ti) = aq,x(0) = 0, 
find u = u minimizing J(u) = J* 1 u 2 dt. 

When the initial conditions become stochastic, we have the following description: At 
the time we start braking, its location and speed fits normal distribution independently. 
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The goal is to stop it with as little energy as possible, simultaneously either close enough 
to expected destination and its speed low enough. 

Its mathematic statement is 


x — u. 

x(0) ~ N(x 0 , 1) 
£(0) ~ N(v o, 1) 
tifixed, 


Cost function is defined as 


Penalty function is defined as 


rti 

J(u) — / u 2 dt, 
Jo 


I(u) = k(x 2 (t i) + x 2 (ti)), 


find u = u minimizing E(J(u) + /(it)). Reorganize the problem and we get the following 
new system: 


{£ = »■*=(£) (317) 

*(0) = *"»~jv i( (^) (J ;)) (3.18) 

tifixed, (3.19) 

rti 

J(u) = E / u 2 — 2k(x\(t) + x 2 (t))dt —> min. (3.20) 

Jo 

The system satisfied injection condition, thus be applied with PMP*. Its generalized 
Hamiltonian is 

K(C, x) = C 1 X 2 + + v(u 2 - 2k(xi{t) + x 2 {t))),C = ^ e Tf it) R 2 , 

First consider abnormal case v — 0: 

h° u (e,x)=£x 2 +£u. 

If maximum of fi°(£*,x) exists, there must be Eft, = 0. This means u = 0, which 
contradicts nontrivial requirement. 

Then consider the normal case v — — 1: 


h u \C,x) = &X2+&U - {u 2 - 2k(xi(t) + X 2 (t))). 

Thus, 

r ^ dh-j 

I dt dxi 

I dQ dhf 1 

{ 42 dt dx 2 4l 

Due to the expected Hamiltonian maximum generated by U, 

Eh-\C,x) = E(£x 2 + Qu - (u 2 - 2k(x l {t) + x 2 {t)))). 



Thus we get 


dEhZ 1 

du 


0 


u(t) = -E£(t) = at + p. 

The optimal control shall be linear. We can easily find the optimal control when put 
the conclusion into origin system. 


4 Conclusion 

The paper is focused on non-feedback control system with stochastic initial condi¬ 
tions through generalizing Pontryagin Maximum Principle in certain degree. Further we 
establish a generalized method to solve problems of this kind. It needs to be explained 
that this method can be just applied to non-feedback systems, the cost expectation of 
which can be minimized. Through the example in Section 3, we found that procedure is 
analogous to the systems without stochasticity. The only difference is that much larger 
calculation is required after expectation is introduced. 

Certainly, the method is with clear limitation. In section 2 we see that PMP* be¬ 
comes unavailable either when the system is without injection conditions or when the 
penalty function is not smooth enough. In terms of the former, a possible solution is to 
further limit the control. In fact, we only need injection condition for the extremal curve 
corresponding to the optimal control. For the latter, smoothing the penalty function can 
be useful. Actually, the system can be divided into several smooth ones as far as the 
non-smooth point set is finite. 

In addition, there is considerable potentiality in this subject. For instance, assume a 
control system can be disturbed instantaneously at certain points. These systems can be 
divided into several systems with stochastic initial conditions and thereupon each can be 
easily solved. Another example is the systems with continuous disturbance. Systems of 
these kinds are yet to be researched in detail. 
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